Software defined networking (sdn) controller orchestration and network virtualization for data center interconnection

ABSTRACT

A data center interconnection (DCI) network may comprise a data center controller (DCC) managing a plurality of data centers (DCs) interconnected by a provider network managed by a network provider controller (NPC). The provider network may be an OpenFlow based software defined networking (SDN) transport network. The DCC may initiate a virtual network service (VNS) negotiation with the NPC to connect the DCs and may specify a network abstraction granularity level. The NPC may respond by computing paths through the provider network accordingly and providing the DCC with one or more virtual networks (VNs). The DCC may compute virtual paths through the VNs and send virtual network element (VNE) connection setup commands to the DCC. The DCC may convert the VNE connection setup commands into network element (NE) commands to setup connections in NEs of the provider network. The DCC and the NPC may perform fault monitoring, detection, and recovery.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/978,867, filed on May 14, 2018, which is a continuation of U.S.patent application Ser. No. 14/245,029, filed on Apr. 4, 2014, whichclaims priority to U.S. Provisional Patent Application 61/808,893, filedApr. 5, 2013 by Young Lee, et. al., and entitled “SDN ControllerOrchestration and Network Virtualization for Data CenterInterconnection”, all of which are incorporated herein by reference asif reproduced in their entirety.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

REFERENCE TO A MICROFICHE APPENDIX

Not applicable.

BACKGROUND

Some data center interconnections (DCIs) may employ pre-allocated staticwide area network (WAN) optical connections. The pre-allocated capacitybetween data center (DC) sites may be designed for peak data rate andmay be underutilized due to fluctuating traffic demands. The staticdesign of DCI may not be suitable for dynamic allocation of newapplications to one DC site out of a plurality of candidate DC siteswith associated WAN bandwidth adjustment. For example, a first DC maywish to migrate some workload and/or data on the fly to a second DC forload balancing or fault recovery, which may not be accomplished in anefficient manner by a static DCI.

Software defined networking (SDN) may be a networking paradigm in whichdata forwarding (e.g. data plane) may be decoupled from controldecisions (e.g. control plane), such as routing, resources and othermanagement functionalities. The decoupling may also allow the data planeand the control plane to operate on different hardware, in differentruntime environments, and/or operate using different models. In an SDNnetwork, network intelligence may be logically centralized insoftware-based controllers (e.g. SDN controllers), which may configurenetwork nodes (e.g. via OpenFlow protocol) and control application datatraffic flow (e.g. compute optimal network paths) in the SDN networkindependent of network hardware. There may be a growing interest fornetwork providers or service providers to adopt SDN over an underlyingnetwork to provide more flexible and scalable transport SDN networkssuitable for DCI.

SUMMARY

An SDN controller orchestration and network virtualization mechanism fordata center interconnection is disclosed herein. In one exampleembodiment, the disclosure includes a network provider controller (NPC)associated with an SDN based transport network. The NPC may create avirtual network (VN) for a data center (DC) client according to avirtual network service (VNS) negotiation initiated by a data centercontroller (DCC) acting on behalf of the client. The VNS negotiationinitiation may comprise network addresses of one or more source DCendpoints, network addresses of one or more destination DC endpoints,and a traffic matrix element comprising traffic characteristics betweenthe source DC endpoints and the destination DC endpoints. The NPC maycompute a plurality of network paths that traverse the network betweenthe source DC endpoints and the destination DC endpoints according tothe traffic matrix element. The NPC may abstractly represent thecomputed network paths in a form of a virtual network topology (VNT) andsend the VNT to the DCC.

In another example embodiment, the disclosure includes a DCC thatemploys an SDN based transport network for transporting data traffic.The DCC may initiate a VNS negotiation with an NPC. The DCC may send aVNS negotiation initiation message to the NPC. The initiation messagemay comprise a network address of a source DC endpoint, networkaddresses of one or more destination DC endpoints, and a networkabstraction granularity level. The DCC may receive a VNS response fromthe NPC. The VNS response may comprise a VNT that comprises virtualnetwork paths that connect the source DC endpoint to the destination DCendpoints. The virtual network paths may abstractly represent paths inthe network according to the network abstraction granularity levelindicated in the VNS negotiation initiation message.

In yet another example embodiment, the disclosure includes a methodimplemented in an NPC. The NPC may negotiate with a DCC to identify aset of endpoints that a DC wishes to connect. The NPC may receive, fromthe DCC, traffic characteristics that need to be supported between theset of endpoints. The NPC may create a VN by initiating a request to avirtual path computation entity of the NPC to compute paths between theset of endpoints. The NPC may allocate network resources to fulfill theVN and may notify the DCC of availability of the VN for transport ofinformation between the endpoints.

These and other features will be more clearly understood from thefollowing detailed description taken in conjunction with theaccompanying drawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of this disclosure, reference is nowmade to the following brief description, taken in connection with theaccompanying drawings and detailed description, wherein like referencenumerals represent like part.

FIG. 1 is a schematic diagram of an example embodiment of an OpenFlowbased transport SDN architecture.

FIG. 2 is a schematic diagram of an example embodiment of a DCI network.

FIG. 3 is a schematic diagram of an example embodiment of a networkelement (NE).

FIG. 4 is a schematic diagram of an example embodiment of a DCconnecting to a provider network that operates in a same administrativedomain.

FIG. 5 is a schematic diagram of an example embodiment of a DCconnecting to a provider network that operates in a differentadministrative domain.

FIG. 6 illustrates an example embodiment of a VNS negotiation method ina DCI network.

FIG. 7A is a schematic diagram of an example embodiment of a VNT.

FIG. 7B is a schematic diagram of another example embodiment of a VNT.

FIG. 8 is a protocol diagram of an example embodiment of a VNSnegotiation method.

FIG. 9 illustrates an example embodiment of a VN connection setup methodin a DCI network.

FIG. 10 is a protocol diagram of an example embodiment of a VNconnection setup method.

FIG. 11 illustrates an example embodiment of a fault monitor, detection,and recovery scenario in a DCI network.

FIG. 12 is a protocol diagram of an example embodiment of a faultrecovery method.

DETAILED DESCRIPTION

It should be understood at the outset that although an illustrativeimplementation of one or more embodiments are provided below, thedisclosed systems and/or methods may be implemented using any number oftechniques, whether currently known or in existence. The disclosureshould in no way be limited to the illustrative implementations,drawings, and techniques illustrated below, including the exemplarydesigns and implementations illustrated and described herein, but may bemodified within the scope of the appended claims along with their fullscope of equivalents.

DCs may provide a wide variety of Internet services includingweb-hosting, e-commerce, enterprise, social networking, and a variety ofgeneral services, such as software as a service, platform as a service,infrastructure as a service, and cloud computing services. Networkvirtualization may be a key to providing many of these Internet servicesand may be increasingly employed within DCs to achieve betterutilization of network resources and more flexible resource allocation.A DC operator may operate a plurality of DCs physically located indifferent geographical sites, where the DCs may be interconnected by atransport network (e.g. provider network) operated by a networkprovider. The network provider may or may not be the same owner as theDC operator. For example, when a provider owns both the provider networkand the DCs, the provider may have full control in optimizing resourceutilization. However, when the network and the DCs are owned bydifferent organizations (e.g. private and public) or a mix oforganizations (e.g. cloud bursting), optimization may be more limited.As such, DC controls and provider network controls may be separated toallow for optimizations in the respective domains. For example, an NPCmay control a provider network and a DCC may control DCs connected bythe provider network. The DCC may be an internal client to the NPC whenthe provider owns both the network and the DCs. Conversely, the DCC maybe an external client to the NPC when the DCs are owned by third party.

Disclosed herein are SDN controller orchestration and networkvirtualization mechanisms for DCI. A DCI network may comprise one ormore DCCs managing a plurality of DCs located at different geographicalsites interconnected via a provider network, which may be an OpenFlowbased SDN transport network. An NPC may be responsible for controllingand managing the provider network, abstracting (e.g. network slicing)the provider network into VNs (e.g. according to clients' servicecontracts and/or requests), and presenting the VNs to one or moreclients who employ the provider network for transport of the clients'data traffic. The DCC may be responsible for coordinating with the NPCto establish and manage connections between the DCs and across multipleapplications. In an embodiment, an NPC may manage and configure networknodes of a provider network and may communicate with a DCC by employingan OpenFlow protocol. In an embodiment, a DCC may initiate a VNSnegotiation with an NPC to connect a plurality of DCs. The DCC may senda VNS negotiation initiation message (e.g. network queryrequest)comprising a set of network addresses for source DC endpointsand destination DC endpoints, a traffic matrix element specifyingtraffic characteristics (e.g. bandwidths, latencies, etc.) between thesource DC endpoints and the destination DC endpoints, and/or a networkabstraction granularity level (e.g. at a network endpoint granularity ora link level granularity). In response to the VNS negotiationinitiation, the NPC may compute network paths through the providernetwork accordingly and may send a VNS response to the DCC in a form ofa VNT (e.g. network resources). The DCC may request the NPC establish aVN connection between a pair of the source and destination endpoints. Inaddition, the DCC and/or the NPC may perform fault monitoring,detection, and/or recovery.

FIG. 1 is a schematic diagram of an example embodiment of an OpenFlowbased transport SDN architecture 100. SDN architecture 100 may comprisea plurality of client controllers (CCs) 110, an NPC 120, and a pluralityof client endpoints (CEs) 130 interconnected by a transport network 140.The transport network 140 may comprise a plurality of network nodes 141interconnected via links 142. The network nodes 141 may be switches,routers, and/or any other devices suitable for forwarding data in thetransport network 140. The links 142 may comprise physical links (e.g.fiber optic links, electrical links), logical links, and or combinationsthereof used to transport data. The transport network 140 may becontrolled by the NPC 120 and the CEs 130 may be controlled by the CCs110. The transport network 140 and the NPC 120 may be operating in asame administrative domain and owned by a network provider or serviceprovider. The CCs 110 and CEs 130 may be owned by one or more clientoperators and may or may not be operating in a same administrativedomain as the NPC 120 and the transport network 140.

NPC 120 may be a virtual machine (VM), a hypervisor, or any other deviceconfigured to manage the transport network 140 via a control data planeinterface (CDPI) 160. The NPC 120 may be a software agent operating onhardware and acting on behalf of a network provider that owns thetransport network 140 (e.g. provider network) and associated topologyresource data. The NPC 120 may be aware and comprise a full view of aphysical topology of the transport network 140. The physical topologymay include all network nodes 141 and links 142 in the transport network140. Topology resource data may comprise connectivity and availableresources (e.g. path latency, bandwidths, capacity, traffic engineering(TE) data, etc.) in the transport network 140. The NPC 120 may maintainand track network resource usage in the transport network 140. Inaddition, the NPC 120 may be configured to provide VNS, such as VNcontrol and management functions to one or more clients by communicatingwith CCs 110 via a control virtual network interface (CVNI) 150. Thetypes of VNS may include VNS establishment and negotiation, VN creation,VN connection setup, fault monitor, fault detection, and fault recovery.

A CC 110 may be a VM, a hypervisor, or any device configured tocoordinate with an NPC 120 to establish and manage network connectionsfor the CEs 130. The CC 110 may be a software agent operating onhardware and acting on behalf of a client operator that owns the CEs130. The CC 110 may initiate VNS requests to the NPC 120 via a CVNI 150.The requests may include VNS establishment and negotiation, VNconnection setup, and/or fault monitor, detection, and recovery. In someexample embodiments, a provider that provides a network (e.g. transportnetwork 140 with an NPC 120) may also be a client that operates the CC110 and CEs 130. In such embodiments, the CCs 110 may be an internalclient controller to the NPC 120. In some other example embodiments, theCC 110 may be an external client controller to the NPC 120 when the CC110 and CEs 130 are operated by third parties.

Each CE 130 may be an egress port of a client site network connecting tothe transport network 140, where the client site networks may bephysically located in different geographical locations. For example, aCE 130 may be a router, a network address translation (NAT) devicegateway, or any network node configured to interface the client sitenetwork to the transport network 140. The client site network maycomprise one or more network sub-domains and each network sub-domain mayor may not employ a same routing protocol. An NPC 120 may identify a CE130 via a network address of the CE 130. For example, a CC 110 maymanually configure and communicate a CE 130′s network address to an NPC120. Alternatively, a link layer discovery protocol (LLDP) may beemployed to discover the CE 130. It should be noted that the CC 110 andthe NPC 120 may support a same network address convention in order forthe NPC 120 to identify the CE 130.

CDPI 160 may be an interface that supports communication between the NPC120 and the network nodes 141 in the transport network 140. CDPI 160 maysupport communication between Open System Interconnection (OSI) layersand between components in the same OSI layer. CVNI 150 may be aninterface (e.g. inter-domain or intra-domain) that supportscommunication between the NPC 120 and CC 110. CVNI 150 may berecursively hierarchical (e.g. supporting multiple client serverlayers). The CDPI 160 and the CVNI 150 may be implemented in variousprotocols, such as an OpenFlow protocol.

The Openflow protocol is a communication protocol that provides a set ofstandard application programming interfaces (APIs) for a remotecontroller to configure, control, and manage network nodes (e.g. networknodes 141) across a network (e.g. network 140) for data forwarding. Forexample, an NPC 120 may compute forwarding paths (e.g. Label SwitchedPath (LSP)) across network 140, configure network nodes 141 with theforwarding paths (e.g. stored in flow tables), and instruct the networknodes 141 to forward data packets according to the computed forwardingpaths. In addition, the NPC 120 may modify, add, and/or delete theforwarding paths dynamically to adapt to network changes (e.g. trafficdemands, network conditions, faults, allocation of new applications,relocations of data, etc.).

In an example embodiment, the SDN architecture 100 may be employed forDCI applications, where CCs 110 may be DCCs and CEs 130 may be DC CEs.FIG. 2 is a schematic diagram of an example embodiment of a DCI network200. Network 200 may comprise a DCC 210, an NPC 220, and a plurality ofDCs 230 interconnected by a transport network 240 comprising a pluralityof interconnecting network nodes 241. The DCC 210, NPC 220, CVNI 250,CDPI 260, network nodes 241, and transport network 240 may besubstantially similar to any of the CCs 110, NPC 120, CVNI 150, CDPI160, network nodes 141, and transport network 140, respectively. Innetwork 200, the DCC 210 may communicate with the NPC 220 to coordinatenetwork connections between the DCs 230, which may be physically locatedin different geographical sites.

A DC 230 may be a facility configured to store data, execute processes,and/or provide network services for clients, such as end users,application service providers, other networks, etc. A DC 230 maycomprise server racks 232, an aggregation server (AS) 233, and a CE 231(e.g. CE 130). The server racks 232 may comprise a plurality ofinterconnected servers (e.g. interconnected by a core network) thatprovides the data storage, process execution, and/or network services. ADC 230 may comprise one or more network sub-domains, where the networksub-domains may employ a same routing protocol or different routingprotocols and may be segregated for security and/or organizationalpurposes. The AS 233 may aggregate and/or distribute communicationsbetween the server racks232 and the CE 231. The CE 231 may connect theDC 230 to the transport network 240.

A DCC 210 may be configured to provide orchestration functions with anNPC 220, provide security for the DC 230, provide NAT, etc. for DCIapplications via a CVNI 250 (e.g. CVNI 150). The orchestration functionsmay include network query (e.g. VNS negotiation), network resourcereservation (e.g. virtual network connection), fault monitor, faultdetection, and fault recovery. The DCC 210 may query the NPC 220 fornetwork resource information (e.g. bandwidth, latency, capacity, etc.)between the CEs 231. Network query may refer to pre-connection query,where the DCC 210 may select an optimal data center location from theconnected DCs 230 for a given workload, data migration, and/orallocation for a new application based on the network resourceavailability reported by the NPC 220. For example, a DCC 210 mayinitiate a VNS negotiation with the NPC 220 to connect a set of DC CEs231 (e.g. by transmitting a location profile with network addresses ofsource DC CEs 231 and destination DC CEs 231). The NPC 220 may alsospecify a traffic requirement matrix between the set of DC CEs 231. Thetraffic requirement matrix may include traffic characteristics (e.g.connection types, bandwidths, latency, Quality of Service (QoS), etc.)between the source DC CEs 231 and the potential destination DC CEs 231.In response to the VNS negotiation initiation, the NPC 220 may estimatenetwork connection resources, determine connection paths (e.g. via apath computation entity (PCE)) between the source DC CEs 231 and thedestination CEs 231 and reply to the DCC 210 with network resourceinformation for each candidate connection. The NPC 220 may create a VN(e.g. instantiate the VNS) and may present the network resourceestimates to the DCC 210 in the form of a VNT. For example, a VNT maycomprise interconnecting virtual network elements (VNEs), which may ormay not correspond directly to the network nodes 241 depending onnetwork abstraction granularity, which may be discussed more fullybelow. It should be noted that the NPC 220 may track the networkresources committed for the network paths in the VNT prior to setting upconnections in the network paths of the VNT.

A DCC 210 may explicitly request the NPC 220 for virtual networkconnections between the CEs 231 after negotiating a VNS instance withthe NPC 220. During virtual network connection setup, the NPC 220 mayallocate and/or reserve network resources (e.g. cross-connect resourcesof a transport network) for network paths that are committed for the VNSinstance. Some of the parameters to support network resource reservationmay include endpoint addresses (e.g. source and destination endpointaddresses), connection types (e.g. point-to-point (P2P),point-to-multipoint (P2MP), multipoint-to-multipoint (MP2MP) sharedpool, etc.), bandwidth information for each connection (e.g. minimumbandwidth, etc.), service duration (e.g. starting time, ending time),protection and/or restoration level (e.g. protection level, 1+1, 1:N,etc.).

A DCC 210 may monitor DC faults in the DCs 230and may request an NPC 220to reroute client traffic from one DC 230 to another DC 230 upondetection of DC faults. Similarly, an NPC 220 may monitor network faultsand request a DCC 210 to migrate data or applications from one DC 230 toanother DC 230 upon detection of network faults. In addition, faultmonitor information may be exchanged between the DCC 210 and the NPC220. The exchange of fault monitor information may be complementary toService Level Agreement (SLA) management and may include performancenotifications (e.g. with consequence actions) from the DCC 210 to theNPC 220.

In an example embodiment, each DC 230 may have a local DCC 210 and theDCCs 210 may form a confederacy or hierarchy (e.g. a logical DCC) tointerface with an NPC 220. The DCC 210 may be a client to the NPC 220.The DCC 210 may be a software agent operating on hardware and operatingon behalf of a DC operator, which may be responsible for coordinatingnetwork resources to meet the requirements of applications hosted in DCs230. The NPC 220 may be a software agent operating on hardware andoperating on behalf of a network provider that provides a transportnetwork 240. The NPC 220 may be responsible for managing the transportnetwork 240.

To support the example embodiments discussed herein, several assumptionsmay be made. The DCC 210 may be aware of all CE231 interfaces that areconnected to the transport network 240. Also, a data plane connectionbetween each CE 231 interface and a corresponding endpoint interface(e.g. a user-network interface (UNI)) of the transport network 240 mayhave been established prior to communications between the DCC 210 andthe NPC 220. Dynamic establishment of a data plane connection may beemployed in some cases to support dynamic attachment to transportnetwork 240, for example via wireless access technologies. In addition,a service contract may be in force between a DC operator that providesthe DCs 230 and a network provider that provides the transport network240. The service contract may set relevant policies regarding operationsof the services available to the DC operator and by extension to the DCC210. The NPC 220 may also be aware of the endpoint interfaces in thetransport network 240 that are connected to DCs 230, operated by the DCoperator, and covered by the service contract. An authenticationmechanism that supports dynamic attachment to the transport network 240(e.g. via wireless access technologies) may be employed in some exampleembodiments. The DCC 210 may have full visibility of (e.g. access toand/or awareness of) each DC 230 under the DCC 210′s control. Thisvisibility may include DC 230 resource information, DC 230 locationinformation, interfaces to transport network 240 and otheruser/application related information.

FIG. 3 is a schematic diagram of an example embodiment of an NE 300,which may act as a CC (e.g. CC 110), an NPC (e.g. NPC 120 and/or 220),or a DCC (e.g. DCC 210) in a DCI network (e.g. network 100 and/or 200).NE 300 may be configured to determine paths, establish connections in aDCI network, and/or perform network fault recovery when acting as anNPC. In an alternative example embodiment, NE 300 may act as anothernode in network 100 and/or 200. NE 300 may be configured to perform VNSnegotiation (e.g. network query), VN connection setup (e.g. networkresource reservation), and/or DC fault recovery when acting as a DCC. NE300 may be implemented in a single node or the functionality of NE 300may be implemented in a plurality of nodes. One skilled in the art willrecognize that the term NE encompasses a broad range of devices of whichNE 300 is merely an example. NE 300 is included for purposes of clarityof discussion, but is in no way meant to limit the application of thepresent disclosure to a particular NE embodiment or class of NEembodiments. At least some of the features/methods described in thedisclosure may be implemented in a network apparatus or component suchas an NE 300. For instance, the features/methods in the disclosure maybe implemented using hardware, firmware, and/or software installed torun on hardware. As shown in FIG. 3, the NE 300 may comprisetransceivers (Tx/Rx) 310, which may be transmitters, receivers, orcombinations thereof. A Tx/Rx 310 may be coupled to plurality ofdownstream ports 320 for transmitting and/or receiving frames from othernodes and a Tx/Rx 310 may be coupled to plurality of upstream ports 350for transmitting and/or receiving frames from other nodes, respectively.A processor 330 may be coupled to the Tx/Rx 310 to process the framesand/or determine which nodes to send the frames to. The processor 330may comprise one or more multi-core processors and/or memory devices332, which may function as data stores, buffers, etc. Processor 330 maybe implemented as a general processor or may be part of one or moreapplication specific integrated circuits (ASICs) and/or digital signalprocessors (DSPs). Processor 330 may comprise a DCI management module333, which may implement a VNS negotiation method 800, a VN connectionsetup method 1000, and/or a fault recovery method 1200 as discussed morefully below. In an alternative embodiment, the DCI management module 333may be implemented as instructions stored in the memory devices 332,which may be executed by processor 330. The memory device 332 maycomprise a cache for temporarily storing content, e.g., a Random AccessMemory (RAM). Additionally, the memory device 332 may comprise along-term storage for storing content relatively longer, e.g., a ReadOnly Memory (ROM). For instance, the cache and the long-term storage mayinclude dynamic random access memories (DRAMs), solid-state drives(SSDs), hard disks, or combinations thereof

It is understood that by programming and/or loading executableinstructions onto the NE 300, at least one of the processor 330 and/ormemory device 332 are changed, transforming the NE 300 in part into aparticular machine or apparatus, e.g., a multi-core forwardingarchitecture, having the novel functionality taught by the presentdisclosure. It is fundamental to the electrical engineering and softwareengineering arts that functionality that can be implemented by loadingexecutable software into a computer can be converted to a hardwareimplementation by well-known design rules. Decisions betweenimplementing a concept in software versus hardware typically hinge onconsiderations of stability of the design and numbers of units to beproduced rather than any issues involved in translating from thesoftware domain to the hardware domain. Generally, a design that isstill subject to frequent change may be preferred to be implemented insoftware, because re-spinning a hardware implementation is moreexpensive than re-spinning a software design. Generally, a design thatis stable that will be produced in large volume may be preferred to beimplemented in hardware, for example in an ASIC, because for largeproduction runs the hardware implementation may be less expensive thanthe software implementation. Often a design may be developed and testedin a software form and later transformed, by well-known design rules, toan equivalent hardware implementation in an ASIC that hardwires theinstructions of the software. In the same manner as a machine controlledby a new ASIC is a particular machine or apparatus, likewise a computerthat has been programmed and/or loaded with executable instructions maybe viewed as a particular machine or apparatus.

FIG. 4 is a schematic diagram of an example embodiment of a DC 430connecting to a provider network 440 that operates in a sameadministrative domain. The DC 430 and the provider network 440 may besubstantially similar to DC 230 and network 240, respectively. The DC430 may comprise server racks 432, an AS 433, and a CE 431. The serverracks 432, the AS 433, and the CE 431 may be substantially similar tothe sever rack 232, AS 233, and CE 231, respectively. Since the DC 430and the network 440 operate in a same administrative domain, one of theDC 430's ports (e.g. CE 431) may be connected to a provider network node441 (e.g. network node 141) and the data plane connectivity between theCE 431 and the provider network node 441 is a network-to-networkinterface (NNI).

FIG. 5 is a schematic diagram of an example embodiment of a DC 530connecting to a provider network 540 that operates in a differentadministrative domain. The DC 530 and the provider network 540 may besubstantially similar to DC 430 and network 440, respectively. However,since the DC 530 and the network 540 operate in different administrativedomains, the DC 530 may comprise an additional provider edge (PE) device534 (e.g. border router, border gateway, etc.) in addition to severracks 532 and an AS 533, where the AS 533 may act as a CE (e.g. CE 431)for the DC 530. The AS 533 may connect to the PE device 534 in the DCpoint of presence (POP) over a UNI as this may be a demarcation pointthat separates respective administrative domains.

FIG. 6 illustrates an example embodiment of a VNS negotiation method 600in a DCI network, which may be substantially similar to DCI network 240.In method 600, an NPC 620 (e.g. NPC 220) may manage a transport network640 (e.g. network 240) comprising interconnecting network nodes 641(e.g. network nodes 241)and a DCC 610 (e.g. DCC 210) may manage aplurality of DCs (e.g. DCs 230) comprising DC CEs 631(e.g. CE 231)connecting to the transport network 640. The NPC 620 may communicatewith the DCC 610 via a CVNI 650 (e.g. CVNI 250). The DCC 610 may beinterested in knowing relevant resource information of the transportnetwork 640 (e.g. for DCI) for transport of data traffic between theDCs. For example, the DCC 610 may initiate a VNS negotiation with theNPC 620 to connect a set of the DC CEs 631 for a DCI application.

The VNS negotiation initiation may include exchanging parameters, suchas a client identifier that identifies a client, a VNS instanceidentifier that identifies a particular instance of a VNS requested bythe client, a connectivity types (e.g. P2P, M2P, M2M), a locationprofile (e.g. source and destination addresses of CEs), a networkabstraction granularity level (e.g. a network endpoint granularity, alink level granularity), a latency constraint, a bandwidth constraint(e.g. minimum bandwidth, excess bandwidth), service duration (e.g. starttime, end time), protection level (e.g. no protection, or otherprotection levels), link type (e.g. dedicated, diverse, etc.), and acost matrix (e.g. price charge information for the requested service).It should be noted that a service contract (e.g. service policies) maybe in place between a client operator who operates the DCC 610 and DCs631 and a service provider who provides the transport network 640 priorto the initiation.

In an example embodiment, a DCC 610 may initiate a VNS negotiation fordedicated P2MP connections between a source CEA631 and destination CEsB, C, and D 631 and may request a VNT. The CEA 631 may comprise a port aconnecting to a transport network 640 via a port x of a network node 1641. The destination DC CE B 631 may comprise a port a connecting to thetransport network 640 via a port y of a network node 7 641. Thedestination DC CE C 631 may comprise a port a connecting to thetransport network 640 via a port y of a network node 3 641. Thedestination DC CE D 631 may comprise a port a connecting to thetransport network 640 via a port y of a network node 8 641. As such, theDCC 610 may construct a location profile specifying a source A.a-1 .xand a list of destinations B.a-7 .y, C.a-3 .y, and D.a-8 .y.

When an NPC 620 receives the VNS negotiation initiation, the NPC 620 mayestimate network resources, compute paths for the requested connectionsaccording to the VNS negotiation initiation parameters. For example, thecomputed paths may include a path 661 between DC CE A 631 and DC CE C631(e.g. dark solid lines in the transport network 640), a path 662between DC CE A 631 and DC CE B 631 (e.g. dark dashed lines in thetransport network 640), and a path 663 between DC CE A 631 and DC CE D631 (e.g. dark dotted lines in the transport network 640). Afterallocating the network resources and computing the network paths, theNPC 620 may abstractly represent the computed network paths in the formof a VNT according to a network abstraction granularity level and maynotify the DCC 610 of the VNT (e.g. available network resource fortransporting data between the requested source CEs 631 and destinationCEs 631).

In an example embodiment, the DCC 610 may request for a networkabstraction granularity level at a network endpoint granularity. In suchan example embodiment, the NPC 620 may respond with a VNT comprisingresource information (e.g. bandwidth, latency, etc.) between the CEs 631and may not include resource information for the internal network nodes641 (e.g. network nodes 141) or links (e.g. links 142) along thecomputed paths. For example, the VNT may comprise a single virtual linkthat represents the path 661 between network nodes 1 and 3, a singlevirtual link that represents the path 662 between network nodes 1 and 7,and a single virtual link that represents the path 663 between networknodes 1 and 7 as shown in VNT 710 of FIG. 7A.

In another example embodiment, the DCC 610 may request for a networkabstraction granularity level at a link level granularity. In suchexample embodiment, the NPC 620 may respond with a VNT comprisingresource information between the DC CEs 631, as well as network nodes641 and links along the computed paths in the transport network 640 asshown in VNT 720 of FIG. 7B. In some other example embodiments, the NPC620 may abstract a portion of the transport network 640 into a singlevirtual node with at least two ports (e.g. one input port and one outputport) while providing network nodes 641 and links of another portion ofthe transport network 640. The formulation of a VNT may depend onvarious factors, which may include a client's request and/or a clientservice agreement and may allow for various granularity levels ofabstraction.

FIG. 8 is a protocol diagram of an example embodiment of a VNSnegotiation 800 method, which may be implemented between a DCC (e.g. DCC610) and an NPC (e.g. 620). Method 800 may be implemented as part of aVNS negotiation in method 600. Method 800 may begin after a servicecontract is established between a network provider and a DC operator.The network provider may provide a provider network (e.g. network 640)and the NPC may manage the provide network. The DC operator may operatea plurality of DCs (e.g. DCs 630) and the DCC may control the DCs andmanage interconnections of the DCs. The DCs may comprise DC endpoints(e.g. DCs 231 or 631) connecting to the provider network. At step 810,the DCC may initiate a VNS negotiation with the NPC by sending a VNSnegotiation initiation message to the NPC to connect the plurality ofDCs. The initiation message may include parameters substantially similarto the VNS negotiation initiation parameters in method 600 describedherein above.

At step 820, upon receiving the network query request from the DCC, theNPC may estimate network resources and compute paths according to theparameters in the VNS negotiation initiation parameters. At step 830,the NPC may send a VNS response message to the DCC in a form of a VNT.The granularity level of the VNT may depend on the initiationparameters. For example, when the initiation specifies a networkendpoint granularity, the VNS response message may comprise resourceinformation (e.g. capacity, cost, latency, etc.) between the DCendpoints, but may not include resources of links (e.g. links 142) ornetwork nodes (e.g. network nodes 141 or 641) along the computed paths.Alternatively, when the initiation specifies a link level granularity,the VNS response message may comprise network resources between the DCendpoints as well as network nodes and links along the computed paths.It should be noted that the messages at steps 810 and 830 may beimplemented as OpenFlow API calls.

FIG. 9 illustrates an example embodiment of a VN connection setup method900 in a DCI network, which may be substantially similar to the DCInetwork 200. In method 900, a DCC 910 (e.g. DCC 610) may request an NPC920 (e.g. NPC 620) to establish a VN connection (e.g. network resourcereservation) between a source DC CE A 931 (e.g. DC CE 631) and adestination DC CE C 931 via a transport network 940 (e.g. network 640)with interconnecting network nodes 941 (e.g. network nodes 641). Itshould be noted that a VNS may be negotiated between the DCC 910 and theNPC 920 prior to the VN connection request via substantially similarmechanisms as described in method600. The DCC 910 may communicate withthe NPC 920 via a CVNI 950 (e.g. CVNI 650). The VN connection requestmay comprise parameters substantially similar to the initiationparameters in method 600 as described herein above. Upon receiving theVN connection request, the NPC 920 may generate commands to setup aconnection 970across the network 940 (e.g. depicted as a dashed blackline) between DC CE A 931 and DC CE C 931 and send a connectionconfirmation to the DCC 910.

In some example embodiments, the DCC 910 may compute a virtual pathbetween the DC CE A 931 and DC CE C 931in a VNT 951 (e.g. VNT 651) andmay send a VNE connection setup command for each VNE in the computedvirtual path to the NPC 920. When a VNE connection setup command is fora VNE that corresponds to a network node 941, the NPC 920 may convertthe VNE connection setup command into a connection command for thenetwork node 941 and may send the connection command to the network node941 via a CDPI 960 (e.g. CDPI 160). When a VNE connection setup commandis for a VNE that corresponds to a sub-network (e.g. a group of networknodes 941) in the transport network 940, the NPC 920 may compute asub-network path across the sub-network, generate connection commandsfor the network nodes 941 in the sub-network path, and send theconnection commands to the network nodes 941 via a CDPI. It should benoted that allocated paths (according to method 600) are shown in greyscale in FIG. 9. Such paths may remain allocated to DCC 910 after setupof connection 970 or released for other operation depending on theembodiment.

FIG. 10 is a protocol diagram of an example embodiment of a VNconnection setup method 1000, which may be implemented between a DCC(e.g. DCC 610 or 910), an NPC (e.g. NPC 620 or 920), and network nodes(e.g. network nodes 641) in a provider network (e.g. network 640 or940). Method 1000 may be implemented as part of a virtual networkconnection setup in method 900. Method 1000 may begin after the DCCcompletes a VNS negotiation with the NPC (e.g. via method 800) for a DCIbetween a source DC CE (e.g. DC CE 631 or 931) and one or moredestination DC CEs. The NPC may provide a VNT to the DCC at the end ofthe VNS negotiation. At step 1010, the DCC may compute paths throughVNEs of the VNT for a connection between the source DC and one of thedestination DCs. At step 1020, the DCC may send a VN connection requestmessage to the NPC. The VN connection request message may comprise oneor more VNE connection setup commands. At step 1030, upon receiving theVN connection request message, the NPC may convert the VNE connectioncommands into commands for setting up the network nodes corresponding tothe VNEs. When a VNE is a sub-network (e.g. abstraction), the NPC maycompute a sub-network path through the sub-network and may generateconnection setup commands for network nodes along the sub-network path.At step 1040, the NPC may send the connection setup commands to thenetwork nodes that correspond to the VNEs or in the computed sub-networkpath. The connection setup command at step 1040 may be shown as a singlestep, but may involve multiple steps and may vary depending on theinterface protocols. At step 1050, after all the connections are setup,the NPC may send a VN connection reply message to the DCC confirming theVN connection request message. It should be noted that the VN connectionrequest message at step 1020, the commands at step 1040, and the VNconnection reply message at step 1050 may be implemented as OpenFlow APIcalls.

FIG. 11 illustrates an example embodiment of a fault monitor, detection,and recovery method 1100 in a DCI network, which may be substantiallysimilar to DCI network 200. In method 1100, a plurality of DCs A, B, C,and D 1130 (e.g. DCs 230) may be connected by a transport network 1140(e.g. network 240) managed by an NPC 1120 (e.g. NPC 220). For example,DC A 1130 may be connected to the transport network 1140 via a networknode 1 1141 (e.g. network node 141) and may be managed by DCC 1110 (e.g.DCC 210), DC C 1130 may be connected to the transport network 1140 via anetwork node 3 1141, DC D 1130 may be connected to the transport network1140 via a network node 8 1141, and DC B 1130 may be connected to thetransport network 1140 via a network node 7. In method 1100, a networkadministrator at DC A1130 may be monitoring for faults in the DCInetwork. Upon detection of a fault, for example, at DC C 1130, thenetwork administrator may initiate an alternative network connectionbetween DC A 1130 to DC B 1130. The DCC 1130 may employ CVNI 1150 (e.g.CVNI 250) to request that the NPC 1120 delete the connection betweennetwork nodes 1 and 3 1141 and connect network nodes 1 and 7 1141 (e.g.via network resource reservation). The NPC 1120 may respond to the DCC1110 by sending an acknowledgement for the connection delete request anda reply for the connection request via CVNI 1150. The alternativenetwork connection between DC A 1130 and DC B 1130 may enable data to bemigrated from DC A 1130 to the DC B 1130.

FIG. 12 is a protocol diagram of an example embodiment of a faultrecovery method 1200, which may be implemented between a DCC (e.g. DCC1110), an NPC (e.g. NPC 1120), and network nodes (e.g. network nodes1141) in a provider network. Method 1200 may be implemented as part of afault recovery in method 1100. Similar to method 1100, the DCC maymanage DCs A, B, and C (e.g. DCs 1130) located at different geographicallocations connecting to the provider network (e.g. network 1140)comprising the network nodes controlled by the NPC, where a VNconnection may be established between DC A and DC C and a VNS may benegotiated for a connection between DC A and DC B. Method 1200 may beginwhen a network administrator detects a fault at DC C and may instructthe DCC to perform fault recovery. At step 1210, the DCC may send a VNconnection deletion request message to the NPC to delete the virtualconnection between DC A and DC C. At step 1220, upon receiving thevirtual connection request, the NPC may convert the VN connection deleterequest message into connection delete commands for network nodes alonga connection path between DC A and DC C. At step 1230, the NPC may sendthe connection delete commands to the network nodes along the connectionpath between DC A and DC C. At step 1240, the NPC may send a connectiondelete acknowledgement message to the DCC. At step 1250, the DCC maysend a VN connection request message to the NPC to setup a VN connectionbetween DC A and DC B. At step 1260, in response to the VN connectionsetup request, the NPC may compute paths and/or convert the VNconnection setup request to connection setup commands for providernetwork nodes to establish the connection between DC A and DC B. At step1270, the NPC may send the connection setup commands to the networknodes along the connection path between DC A and DC B. At step 1280,after the connection setup is completed, the NPC may send a VNconnection reply message to the DCC to confirm the setup of theconnection. Subsequently, data may be migrated from DC A to DC B. Itshould be noted that the VN connection setup steps 1250-1280 may besubstantially similar to method 1000. In addition, the messages betweenthe NPC and the DCC, as well as messages between the NPC and the networknodes in method 1200 may be implemented as OpenFlow API calls.

In an example embodiment of an optical transport network, a use case asdefined in Open Networking Foundation (ONF)document Optical TransportUse cases by Piotr Myslinksi, et. al, which is incorporated herein byreference, may include a VNS establishment, a VN connection, a VNEconnection, and fault monitor, detection, and recovery. The use casescenario may concern pre-virtual network connection information exchangeand its workflow. The exchange process may involve negotiation of a VNSbetween a DCC(e.g. DCC 210) and an NPC (e.g. NPC 210) within theconfines of established business agreement between a DC operator thatoperates the DCC and a service provider that operates the NPC. Thisprocess may provide two objectives. The first objective may be to verifythe established business agreement between the DC operator and theservice provider. The second objective may be to automate businessnegotiation and/or re-negotiation, and VNS creation processes. Acustomer may define the need for a VN within the provider's transportnetwork and then may contact the service provider to negotiate acontract for a VNS. The customer may provide a traffic demand matrix aspart of its VNS request and the service provider may compute and presentto the customer one or more VNs that support the requested objectiveincluding bandwidth. Each VN may comprise one or more VNEs, which may beinterconnected by virtual links and may be characterized by means of anumber of attributes (e.g. committed bandwidth, excess bandwidth,latency, maximum number of supported connections, etc.).

The negotiation of a VNS may be per VNS instance. The VNS may beinstantiated before the first VN connection is to be set up. Theinstantiation of the VNS may result in allocation of committed resourcesof the VN. The resources in the network path in the sub-network (e.g.cross connect resources) may only be allocated when a VNE connectionsetup is performed.

In an example embodiment of a VNS negotiation initiation in the opticaltransport network use case, the VNS negotiation may be between a DCC andan NPC. An actor that initiates a VNS negotiation may be the DCC. As aclient to a transport network, the DCC may be interested in knowingrelevant transport network resource information, especially in light ofDCI. Initially, the DCC may negotiate with the NPC to identify a set ofendpoints it wishes to connect. As part of the negotiation, the DCC mayalso express traffic characteristics that need to be supported betweenthe set of endpoints such as traffic demand (e.g. bandwidth), and QoSrequirements associated with DC endpoint interface pairs.

To allow negotiation to take place, the correspondence between DCendpoint interface identifiers (DC EPID) and provider network endpointinterface identifiers (PN EPID) may be established. This may beaccomplished using a manual process (e.g. exchanging identifiers betweenDC operations and a provider network operation personnel) or it may beautomated (e.g. using LLDP) to exchange endpoint identifiers at UNIs. Ifthe endpoint interfaces are under SDN control (e.g. OpenFlow) thisexchange may be done using PACKET OUT and PACKET IN messages. By thisexchange both the DCC and the NPC may acquire the association betweenthe DC EPID and the PN EPID. Usage of protocols such as the LLDP or theOpenFlow protocol may require some authentication of the exchangedinformation or of the end-points that exchange the information.

During a VNS service negotiation with the DCC, the NPC may create aresponse to the DCC with an associated VN. If the DC endpoints areidentified in a CVNI (e.g. CVNI 250) using the DC EPIDs, the NPC maytranslate each of the DC EPIDs to the associated PN EPIDs beforeproceeding to process the VNS negotiation request. If the endpoints areidentified by the DCC using an identifier (ID) pair (DC EPID, PN EPID),the NPC may not access a translation service and may process the requestusing its own identifier from each ID pair. The NPC may provide a VN inresponse to the DCC. The granularity of the VN may have beenpre-negotiated by a mutual business contract and policy and may beexpressed in terms of VNEs and virtual links and their identifiers. Inorder to provide a relevant VN, the NPC may initiate a request to avirtual network computation entity of the NPC and may determinefeasibility of the request. The corresponding result may be sent by theNPC to the DCC.

A VNS negotiation initiation and/or response may include a set ofminimum parameters, such as a VNS instance identifier, a traffic matrixelement, location information element, and a VN description element. TheVNS instance identifier may identify a particular instance of a VNS. Thetraffic matrix element may describe traffic demand and other QoSinformation associated with DC endpoint interface pairs (e.g, latency).The traffic matrix element may include a connectivity type, a bandwidthtype, and the connectivity's directionality. For example, a connectivitytype may include P2P, P2MP, multipoint-to-multipoint (MP2MP) (e.g. anycast), or rooted-multipoint. A bandwidth type may include committedbandwidth resource, excess bandwidth resource, or shared pool. Eachconnectivity type may include a directionality(e.g. unidirectional,bidirectional). The location information element may describe DCendpoint interfaces (e.g. CEs 231) associated with a connectivity type.For unidirectional connectivity, a source list and a destination listmay need to be distinguished. The VN description element may describeVNEs and virtual links and their identifiers that belong to the VN.

Following negotiation of the VNS, the NPC may allocate network resourcesneeded to fulfill the agreed upon VN. When these resources have beenallocated, the NPC may notify the DCC of the availability of the VN fortransport of information between DC endpoints. The resources in anetwork path in a sub-network (e.g., cross connect resources) may onlybe allocated when a VNE connection setup is performed, as discussedherein.

In an example embodiment of a VN connection setup in the opticaltransport network use case, a DCC may receive a service request from oneof its business applications or from its network engineering functionand the DCC's path computation function may compute a path through a VNbased on a VNS establishment process as described herein above. Aftercompleting the path computation, the DCC may issue VNE connection set upcommands and may send the commands via a CVNI (e.g. CVNI 250) to theNPC. An actor of a VNE connection command may be the DCC. The DCC maysend a VNE connection set up command to each VNE in the path computed bya DCC's path computation function, and each VNE sets up the connectionin the NE or in a sub-network represented by the VNE. The DCC may be infull control of the path through its VNEs and virtual links and/ortunnels. When sending VNE connection commands to each VNE, it is theDCC's responsibility to provide relevant parameters to each VNE.

Any change of existing active VNE connections, such as bandwidthmodification, may also be supported. There may be bandwidth modificationor other changes to a VN state (e.g., capacity, delay, etc.) that fallsoutside the constraints of the existing VNS agreement and maynecessitate a renegotiated VNS agreement. A renegotiation may be donebased on a same VNS instance with modified attributes.

An actor of a VNE connection confirmation process may be the NPC. Uponthe receipt of the VNE connection setup commands from the DCC, a VNEconnection setup command may be converted into a command to set up aconnection in an NE represented by the VNE, or the NPC's pathcomputation function may be triggered on a VNE representing asub-network in the provider network to compute a path through thesub-network that may comply with the VNE connection setup command.Following path computation on each VNE, the NPC may be responsible forsetup of a connection in each NE within the computed path of thesub-network.

In some example embodiments, the VNE connection setup commands may notbe able to be successfully completed. A sub-network outage, for example,may cause a setup command to fail due to insufficient bandwidth. Afailure response may include enough information to permit a client toevaluate alternatives and may initiate new VNE connection commands. Therecovery action may be done within constraints of an established servicelevel agreement (SLA). Any recovery action beyond the SLA level maynecessitate a renegotiated VNS agreement only if it has been agreed.

Multiple pre-negotiated policies may exist between an NPC and a DCC,their terms may depend on VNS contract, and the relationship of the NPCand DC administrative authorities. In an example embodiment, an NPC anda DCC may be operated by a same service provider, the NPC may providemore detailed information on the VNS. On the other hand, when an NPC anda DCC belong to different administrative entities, the NPC may hidenetwork details and may apply additional security measurement to protectthe network from potential attacks.

VNE connection commands and/or confirmations may be supported via aCVNI. VNE connection commands may include VNE connection set up, modify,and tear down commands. VNE connection commands and/or confirmations maysupport a minimum set of parameters for each VNE. The parameters mayinclude a VNE identifier, a VNE connection description element, alocation information element, a protection and/or restoration element,and a service duration element. The VNE identifier may identify a VNE.The VNE connection description element may describe a connectivity type,a bandwidth type and the connectivity's directionality, and otherparameters for the VNE. For example, the connectivity type for each VNEconnection instance may be expressed as P2P, P2MP, MP2MP, androoted-multipoint. The bandwidth type for each VNE connection instancemay be expressed as committed bandwidth resources, excess bandwidthresources, shared pool, or other bandwidth types. The directionality foreach VNE connection instance may be expressed as unidirectional orbidirectional. The location information element may describe VNEconnection endpoint interfaces (e.g. virtual links, physical links, orlogical ports) associated with a VNE connection. For unidirectional VNEconnections, a source list may be distinguished from a destination list.For rooted-multipoint VNE connections, a root, leaf, and leaf groupports may be described separately. The protection and/or restorationelement may describe diversity criteria requested and/or applied foreach VNE connection (e.g. diversity requirements relative to other VNEconnections). The service duration element may specify a VNE connectionduration in terms of a begin time and an end time. It should be notedthat the service duration element may be optional. When the serviceduration element is not present, the end time may be undetermined andthe connection for the VNS may begin immediately.

In an example embodiment of a fault monitoring, detection, and recoveryin the optical transport network use case, fault monitoring informationmay be exchanged between a DCC and an NPC. This function may becomplementary to SLA management and may be related to fault andperformance notifications to the NPC with consequent actions. There maybe two types of faults. The first type may be associated with VNresources failing, partial failing, or degrading. The second type may beassociated with faults in VN connections. If a virtual link is protectedor restorable as part of an SLA, then the NPC may hold off any failureand/or degradation report to the DCC while the NPC is protecting and/orrestoring the virtual link. Some examples of faults or performanceissues that may affect an SLA may include an irrecoverable fault in theoptical transport network or service degradation. An irrecoverable faultmay be caused by an ineffective protection and/or restoration in anoptical transport network, an unmanaged failure in a backup facility,insufficient and/or pre-empted resources, or no deployment ofsurvivability mechanisms. Service degradation may occur when an opticaltransport for DCI connection still exists, but the service is degraded,for example, a guaranteed bandwidth may not be satisfied, a requiredlatency may not be satisfied (e.g. may be augmented with a currentlatency value), an increase in bit error rate (BER), and/or a decreasein effective bit rate (EBR). Some consequent actions resulting fromfault and performance notifications may include an NPC determining totake appropriate consequent actions within the optical transport network(e.g. on a case-by-case basis) based upon associated network operatorpolicies or triggering execution (e.g. through a DCC) of appropriateconsequent action policies for clients or customers.

The present disclosure may benefit both DC operators and serviceproviders. Such benefits may include improving optical transport networkcontrol and management flexibility(e.g. the ability to deploy thirdparty client management and/or control systems) and development of newservice offered by network virtualization. The CVNI may enableprogrammatic and virtual control of optical transport networks byallowing applications to have greater visibility and control overconnections carrying the applications' data, and may enable monitoringand protection of the connections (e.g. subject to an operator policy).

At least one embodiment is disclosed and variations, combinations,and/or modifications of the embodiment(s) and/or features of theembodiment(s) made by a person having ordinary skill in the art arewithin the scope of the disclosure. Alternative embodiments that resultfrom combining, integrating, and/or omitting features of theembodiment(s) are also within the scope of the disclosure. Wherenumerical ranges or limitations are expressly stated, such expressranges or limitations should be understood to include iterative rangesor limitations of like magnitude falling within the expressly statedranges or limitations (e.g. from about 1 to about 10 includes, 2, 3, 4,etc.; greater than 0.10 includes 0.11, 0.12, 0.13, etc.). For example,whenever a numerical range with a lower limit, R₁, and an upper limit,R_(u), is disclosed, any number falling within the range is specificallydisclosed. In particular, the following numbers within the range arespecifically disclosed: R=R₁+k*(R_(u)−R₁), wherein k is a variableranging from 1 percent to 100 percent with a 1 percent increment, i.e.,k is 1 percent, 2 percent, 3 percent, 4 percent, 7 percent, . . . , 70percent, 71 percent, 72 percent, . . . , 97 percent, 96 percent, 97percent, 98 percent, 99 percent, or 100 percent. Moreover, any numericalrange defined by two R numbers as defined in the above is alsospecifically disclosed. Unless otherwise stated, the term “about” means±10% of the subsequent number. Use of the term “optionally” with respectto any element of a claim means that the element is required, oralternatively, the element is not required, both alternatives beingwithin the scope of the claim. Use of broader terms such as comprises,includes, and having should be understood to provide support fornarrower terms such as consisting of, consisting essentially of, andcomprised substantially of. Accordingly, the scope of protection is notlimited by the description set out above but is defined by the claimsthat follow, that scope including all equivalents of the subject matterof the claims. Each and every claim is incorporated as furtherdisclosure into the specification and the claims are embodiment(s) ofthe present disclosure. The discussion of a reference in the disclosureis not an admission that it is prior art, especially any reference thathas a publication date after the priority date of this application. Thedisclosure of all patents, patent applications, and publications citedin the disclosure are hereby incorporated by reference, to the extentthat they provide exemplary, procedural, or other details supplementaryto the disclosure.

While several embodiments have been provided in the present disclosure,it should be understood that the disclosed systems and methods might beembodied in many other specific forms without departing from the spiritor scope of the present disclosure. The present examples are to beconsidered as illustrative and not restrictive, and the intention is notto be limited to the details given herein. For example, the variouselements or components may be combined or integrated in another systemor certain features may be omitted, or not implemented.

In addition, techniques, systems, subsystems, and methods described andillustrated in the various embodiments as discrete or separate may becombined or integrated with other systems, modules, techniques, ormethods without departing from the scope of the present disclosure.Other items shown or discussed as coupled or directly coupled orcommunicating with each other may be indirectly coupled or communicatingthrough some interface, device, or intermediate component whetherelectrically, mechanically, or otherwise. Other examples of changes,substitutions, and alterations are ascertainable by one skilled in theart and could be made without departing from the spirit and scopedisclosed herein.

1. A method by a network provider controller, comprising: receiving,from a client controller, a set of endpoints managed by the clientcontroller that are to be connected and traffic characteristics thatneed to be supported between the set of endpoints; establishing avirtual network (VN) in response to the client controller; allocatingnetwork resources for the VN; and notifying the client controller ofavailability of the VN between the endpoints.
 2. The method of claim 1,wherein the traffic characteristics comprise one or more of bandwidthand Quality of Service (QoS).
 3. The method of claim 1 furthercomprising: receiving virtual network element (VNE) connection setupcommands for a path through the VN from the client controller; andconverting one of the VNE connection setup commands into a command toconfigure a network element (NE) represented by a VNE in the path. 4.The method of claim 1 further comprising: receiving virtual networkelement (VNE) connection setup commands for a path through the VN fromthe DCC, wherein a VNE in the path represents a sub-network in aprovider network; triggering a path computation on the VNE; andconfiguring each network element (NE) within the computed path of thesub-network in the provider network.
 5. The method of claim 1 furthercomprising exchanging fault monitoring information with the clientcontroller, wherein the fault monitoring information comprises a VNresources failing, a VN resources partial failing, a VN resourcesdegrading, or faults in virtual network connections.
 6. The method ofclaim 1, wherein the client controller is a data center (DC) controller,the end points are endpoints in one or more DCs associated with the DCcontroller.
 7. A network provider controller, comprising: a transceiver;a memory storing instructions for execution by one or more processors,:one or more processors coupled to the transceiver that execute computerinstructions stored in the memory to cause the network providercontroller to: receive, from the client controller, a set of endpointsthat are managed by the client controller that are to be connected andtraffic characteristics that need to be supported between the set ofendpoints; provide a virtual network (VN) in response to the clientcontroller; allocate network resources to fulfil the VN; and send one ormore messages to notify the client controller of availability of the VNfor transport of information between the endpoints.
 8. The networkprovider controller of claim 7, wherein the VN is computed according toa granularity of the VN, the VN is expressed of virtual network elementsand virtual links.
 9. The network provider controller of claim 7,wherein the network provider controller is further configured to:receive virtual network element (VNE) connection setup commands for apath through the VN from the client controller; and convert one of thecommands into a command to configure a network element (NE) representedby a VNE in the path.
 10. The network provider controller of claim 7,wherein: the network provider controller is further configured to:receive a VN connection delete request message from the clientcontroller; and to convert the VN connection delete request message intoconnection delete commands for network nodes along a connection path inthe VN.
 11. A client controller, comprising: at least one processor; anda memory coupled to the at least one processor and storing instructionsfor execution by the at least one processor to cause the clientcontroller to: compute a virtual path through a virtual network (VN),wherein the VN is provided by a network provider controller; and sendvirtual network element (VNE) connection setup commands for VNEs in thecomputed virtual path to the network provider controller.
 12. The clientcontroller of claim 11, wherein the processor is further configured toexecute the instructions to cause the network provider controller to:initiate a virtual network service (VNS) negotiation with the networkprovider controller by sending a VNS negotiation initiation message tothe network provider controller, wherein the a VNS negotiationinitiation message comprises a set of endpoints associated with the VN,the endpoints are managed by the client controller.
 13. The clientcontroller of claim 12, wherein the VNS request further comprises atraffic matrix element comprising traffic characteristics between theendpoints.
 14. The client controller of claim 13, wherein the trafficmatrix element comprises a connectivity type parameter and wherein theconnectivity type parameter comprises a point-to-point (P2P) connection,a point-to-multipoint (P2MP) connection, a multipoint-to-multipoint(M2MP) connection, a rooted multipoint, or combinations thereof.
 15. Theclient controller of claim 11, wherein the processor is furtherconfigured to execute the instructions to cause the network providercontroller to: sent, to the network provider controller, a fault messagethat indicates a detected fault in a remote data center associated withthe client controller, the client controller being a data centercontroller.